Example: Multilabel classification¶
This example shows how to use ATOM to solve a multilabel classification problem.
The data used is a synthetic dataset created using sklearn's make_multilabel_classification function.
Load the data¶
In [1]:
Copied!
# Import packages
import pandas as pd
from atom import ATOMClassifier
from sklearn.datasets import make_multilabel_classification
# Import packages
import pandas as pd
from atom import ATOMClassifier
from sklearn.datasets import make_multilabel_classification
In [2]:
Copied!
# Create data
X, y = make_multilabel_classification(n_samples=300, n_classes=3, random_state=1)
# Create data
X, y = make_multilabel_classification(n_samples=300, n_classes=3, random_state=1)
Run the pipeline¶
In [3]:
Copied!
# Note that for multioutput tasks, you must specify the `y` keyword
atom = ATOMClassifier(X, y=y, verbose=2, random_state=1)
# Note that for multioutput tasks, you must specify the `y` keyword
atom = ATOMClassifier(X, y=y, verbose=2, random_state=1)
<< ================== ATOM ================== >> Configuration ==================== >> Algorithm task: Multilabel classification. Dataset stats ==================== >> Shape: (300, 23) Train set size: 240 Test set size: 60 ------------------------------------- Memory: 51.73 kB Scaled: False Outlier values: 29 (0.5%)
In [4]:
Copied!
# Show the models that natively support multilabel tasks
atom.available_models()[["acronym", "model", "native_multilabel"]]
# Show the models that natively support multilabel tasks
atom.available_models()[["acronym", "model", "native_multilabel"]]
Out[4]:
| acronym | model | native_multilabel | |
|---|---|---|---|
| 0 | AdaB | AdaBoost | False |
| 1 | Bag | Bagging | False |
| 2 | BNB | BernoulliNB | False |
| 3 | CatB | CatBoost | False |
| 4 | CatNB | CategoricalNB | False |
| 5 | CNB | ComplementNB | False |
| 6 | Tree | DecisionTree | True |
| 7 | Dummy | Dummy | False |
| 8 | ETree | ExtraTree | True |
| 9 | ET | ExtraTrees | True |
| 10 | GNB | GaussianNB | False |
| 11 | GP | GaussianProcess | False |
| 12 | GBM | GradientBoostingMachine | False |
| 13 | hGBM | HistGradientBoosting | False |
| 14 | KNN | KNearestNeighbors | True |
| 15 | LGB | LightGBM | False |
| 16 | LDA | LinearDiscriminantAnalysis | False |
| 17 | lSVM | LinearSVM | False |
| 18 | LR | LogisticRegression | False |
| 19 | MLP | MultiLayerPerceptron | True |
| 20 | MNB | MultinomialNB | False |
| 21 | PA | PassiveAggressive | False |
| 22 | Perc | Perceptron | False |
| 23 | QDA | QuadraticDiscriminantAnalysis | False |
| 24 | RNN | RadiusNearestNeighbors | True |
| 25 | RF | RandomForest | True |
| 26 | Ridge | Ridge | True |
| 27 | SGD | StochasticGradientDescent | False |
| 28 | SVM | SupportVectorMachine | False |
| 29 | XGB | XGBoost | False |
In [5]:
Copied!
atom.run(models=["LDA", "RF"], metric="recall_weighted")
atom.run(models=["LDA", "RF"], metric="recall_weighted")
Training ========================= >> Models: LDA, RF Metric: recall_weighted Results for LinearDiscriminantAnalysis: Fit --------------------------------------------- Train evaluation --> recall_weighted: 0.9124 Test evaluation --> recall_weighted: 0.8351 Time elapsed: 0.037s ------------------------------------------------- Time: 0.037s Results for RandomForest: Fit --------------------------------------------- Train evaluation --> recall_weighted: 1.0 Test evaluation --> recall_weighted: 0.8763 Time elapsed: 0.170s ------------------------------------------------- Time: 0.170s Final results ==================== >> Total time: 0.269s ------------------------------------- LinearDiscriminantAnalysis --> recall_weighted: 0.8351 RandomForest --> recall_weighted: 0.8763 !
In [6]:
Copied!
# Note that non-native multioutput models use a meta-estimator wrapper
print(f"Estimator for LDA is: {atom.lda.estimator}")
print(f"Estimator for RF is: {atom.rf.estimator}")
# Note that non-native multioutput models use a meta-estimator wrapper
print(f"Estimator for LDA is: {atom.lda.estimator}")
print(f"Estimator for RF is: {atom.rf.estimator}")
Estimator for LDA is: ClassifierChain(base_estimator=LinearDiscriminantAnalysis(), random_state=1) Estimator for RF is: RandomForestClassifier(n_jobs=1, random_state=1)
Add custom multilabel models¶
To use your own meta-estimator with custom parameters, add it as a custom model. It's also possible to tune the hyperparameters of this custom meta-estimator.
In [7]:
Copied!
from atom import ATOMModel
from sklearn.multioutput import ClassifierChain
from sklearn.linear_model import LogisticRegression
from optuna.distributions import CategoricalDistribution, IntDistribution
custom_model = ATOMModel(
estimator=ClassifierChain(LogisticRegression(), cv=3),
name="chain",
needs_scaling=True,
native_multilabel=True,
)
atom.run(
models=custom_model,
n_trials=5,
ht_params={
"distributions": {
"order": CategoricalDistribution([[0, 1, 2], [2, 1, 0], [1, 2, 0]]),
"base_estimator__max_iter": IntDistribution(100, 200, step=10),
"base_estimator__solver": CategoricalDistribution(["lbfgs", "newton-cg"]),
}
},
)
from atom import ATOMModel
from sklearn.multioutput import ClassifierChain
from sklearn.linear_model import LogisticRegression
from optuna.distributions import CategoricalDistribution, IntDistribution
custom_model = ATOMModel(
estimator=ClassifierChain(LogisticRegression(), cv=3),
name="chain",
needs_scaling=True,
native_multilabel=True,
)
atom.run(
models=custom_model,
n_trials=5,
ht_params={
"distributions": {
"order": CategoricalDistribution([[0, 1, 2], [2, 1, 0], [1, 2, 0]]),
"base_estimator__max_iter": IntDistribution(100, 200, step=10),
"base_estimator__solver": CategoricalDistribution(["lbfgs", "newton-cg"]),
}
},
)
Training ========================= >> Models: chain Metric: recall_weighted Running hyperparameter tuning for ClassifierChain... | trial | order | base_estimator__max_iter | base_estimator__solver | recall_weighted | best_recall_weighted | time_trial | time_ht | state | | ----- | --------- | ------------------------ | ---------------------- | --------------- | -------------------- | ---------- | ------- | -------- | | 0 | [2, 1, 0] | 130 | lbfgs | 0.8831 | 0.8831 | 2.813s | 2.813s | COMPLETE | | 1 | [1, 2, 0] | 150 | newton-cg | 0.9091 | 0.9091 | 2.184s | 4.997s | COMPLETE | | 2 | [2, 1, 0] | 170 | newton-cg | 0.8701 | 0.9091 | 0.085s | 5.082s | COMPLETE | | 3 | [1, 2, 0] | 200 | newton-cg | 0.9221 | 0.9221 | 0.084s | 5.166s | COMPLETE | | 4 | [2, 1, 0] | 100 | newton-cg | 0.8701 | 0.9221 | 0.078s | 5.244s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 3 Best parameters: --> order: [1, 2, 0] --> base_estimator__max_iter: 200 --> base_estimator__solver: newton-cg Best evaluation --> recall_weighted: 0.9221 Time elapsed: 5.244s Fit --------------------------------------------- Train evaluation --> recall_weighted: 0.9021 Test evaluation --> recall_weighted: 0.866 Time elapsed: 0.101s ------------------------------------------------- Time: 5.345s Final results ==================== >> Total time: 5.397s ------------------------------------- ClassifierChain --> recall_weighted: 0.866
Analyze the results¶
In [8]:
Copied!
thresholds = atom.rf.get_best_threshold()
print(f"Best threshold per target column: {thresholds}")
thresholds = atom.rf.get_best_threshold()
print(f"Best threshold per target column: {thresholds}")
Best threshold per target column: [0.7, 0.69, 0.63]
In [9]:
Copied!
atom.rf.evaluate(threshold=thresholds)
atom.rf.evaluate(threshold=thresholds)
Out[9]:
accuracy 0.5667 ap 0.8893 f1_weighted 0.7274 jaccard_weighted 0.6271 precision_weighted 0.8269 recall_weighted 0.6495 auc 0.9213 Name: RF, dtype: float64
In [10]:
Copied!
# Use the target parameter in plots to specify which target column to use
atom.plot_roc(target=2)
# Use the target parameter in plots to specify which target column to use
atom.plot_roc(target=2)
In [11]:
Copied!
# When the target parameter also specifies the class, use format (column, class)
atom.plot_probabilities(models="chain", target=(2, 1))
# When the target parameter also specifies the class, use format (column, class)
atom.plot_probabilities(models="chain", target=(2, 1))
In [12]:
Copied!
with atom.canvas(figsize=(900, 600)):
atom.plot_calibration(target=0)
atom.plot_calibration(target=1)
with atom.canvas(figsize=(900, 600)):
atom.plot_calibration(target=0)
atom.plot_calibration(target=1)